Ronneberger, Fischer, & Brox (2015) – „[…] presents a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently” (w naszym przypadku nie to jest najmocniejszą stroną tego algorytmu).
Ioffe & Szegedy (2015), p. 5 – sectons 3.3 and 3.4: batch normalization enables higher learning rates and regularizes the model.
Milletari, Navab, & Ahmadi (2016), p.6 – ,,[…] not need to assign weights to samples of different classes to establish the right balance between foreground and background voxels”
Masters & Luschi (2018), Abstract – While the use of large mini-batches increases the available computational parallelism, small batch training has been shown to provide improved generalization performance and allows a significantly smaller memory footprint, which might also be exploited to improve machine throughput.
Zhang, Bengio, Hardt, Recht, & Vinyals (2021), Abstract – Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance.
Multi-label balanced loss function:
mbdice_loss <- function(y_true, y_pred, epsilon=0) {
intersection = keras::k_sum(y_true*y_pred, axis=c(2,3))
union = keras::k_sum(y_true*y_true+y_pred*y_pred, axis=c(2,3))
dice = keras::k_mean(2*intersection/(union+epsilon), axis=c(2,1))
return(1 - dice)
}
See section 3 „Dice loss layer” in Milletari, Navab, & Ahmadi (2016), p.6 for 2-label definition.
training_labels <- c(
loss = "Multilabel Balanced Dice Loss",
sdice = "Multilabel Dice Score",
background_dice = "Background Dice Score",
vsat_dice = "VAT Dice Score",
scat_dice = "SAT Dice Score"
)
mdice = "run/2021-03-10/unet_2d_mdice/fat_768x384x1/unet_2d_mdice_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24.csv"
tb <- readr::read_csv(mdice)
hist <- to_history(tb)
early_stoppings(hist, tolerance = 0.005) %>% arrange(metric, epoch)
plot_history(
hist,
lookup_table = training_labels,
segments = early_stoppings(hist, tolerance = 0.005),
base_size = 12
) +
plot_annotation(
title = 'The surprising truth about training UNets neural networks',
subtitle = 'These 5 plots will reveal yet-untold secrets about our beloved data-set',
caption = 'UNet NN: unet_2d_mdice_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24'
)
mdice = "run/2021-03-02/unet_2d_mdice/fat_768x384x1/unet_2d_mdice_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24_DR40.csv"
tb <- readr::read_csv(mdice)
hist <- to_history(tb)
plot_history(
hist,
lookup_table = training_labels,
segments = early_stoppings(hist, tolerance = 0.005),
base_size = 12
) +
plot_annotation(
title = 'The surprising truth about training UNets neural networks',
subtitle = 'These 5 plots will reveal yet-untold secrets about our beloved data-set',
caption = 'UNet NN: unet_2d_mdice_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24_DR20'
)
mdice_ag = "run/2021-03-01/unet_2d_mdice_ag/fat_768x384x1/unet_2d_mdice_ag_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24.csv"
tb_ag <- readr::read_csv(mdice_ag)
hist_ag <- to_history(tb_ag)
plot_history(
hist_ag,
lookup_table = training_labels,
segments = early_stoppings(hist_ag, tolerance = 0.005),
base_size = 14
) +
plot_annotation(
title = 'The surprising truth about training UNets neural networks',
caption = mdice_ag
)
mdice_ag = "run/2021-03-02/unet_2d_mdice_ag/fat_768x384x1/unet_2d_mdice_ag_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24_DR20.csv"
tb_ag <- readr::read_csv(mdice_ag)
hist_ag <- to_history(tb_ag)
plot_history(
hist_ag,
lookup_table = training_labels,
segments = early_stoppings(hist_ag, tolerance = 0.005),
base_size = 12
) +
plot_annotation(
title = 'The surprising truth about training UNets neural networks',
subtitle = 'These 5 plots will reveal yet-untold secrets about our beloved data-set',
caption = 'UNet NN: unet_2d_mdice_ag_AdaDelta_fat_768x384x1_N204_E24_L4_FBL24_DR20'
)